Incomplete Trace Data and Trace Driven Simulation

| We illustrate the sensitivity of trace driven simulation results to the completeness of the input trace data. This is done by comparing cache simulation results obtained using traces similar to many found in the published literature with those obtained using a hardware monitor which we have developed. Cache miss rates from single process traces without operating system references are shown to be in error by as much as a factor of 50. Published synthetic multiprogram workloads are then shown to result in simulated cache miss rates which are in error by factors that range from 0:9 to 110. Finally, it is shown that errors of this magnitude yield misleading performance estimates. Introduction and Background Trace driven simulation has been used for many years to predict the performance of computer systems. This method of simulation may provide useful estimates of system performance, but is dependent on the quality of both the simulation model and the input trace data. While the generation of complete address traces has been the subject of previous work [ASH86], little has been done to compare simulation results using them with those obtained using the incomplete and suspect [Gai91] traces used in published studies [OMB91, WHK91]. We have constructed a hardware monitor capable of producing long and complete traces on a variety of hardware and software platforms. Using these highly accurate traces we quantify the error in system performance predictions which result from the use of incomplete trace data. For this study, we de ne complete address traces to be those traces containing all CPU generated references including those produced by interrupts, system calls, exception handlers, other supervisor activities, and user processes. First, we describe our trace gathering technique and its advantages over other reported methods. We also discuss and quantify the dilation e ects present in our traces. We then examine traces resulting from the execution of a single process by itself. We demonstrate that not only does the cache miss rate vary widely from one portion of a program to another (as previously reported in [BKW90]), but that the majority of cache misses in large sections of the program are due to system code such as interrupts and system calls and that user code is insigni cant in these sections. We show that miss rates predicted from our complete traces are as much as 50 times higher than those predicted from \user code only" traces, similar to those used in many previously published studies. We then turn to multiprogram trace generation. Using complete multiprogram traces collected with our monitor we demonstrate that proposed synthetic trace generation methods for multiprogram workloads may give results little better than those obtained by simply concatenating a collection of single-process traces together. Indeed, we show that more complex schemes which attempt to accurately model context switch intervals are no better than simpler methods and may result in cache miss rate predictions in error by as much as a factor of 110. Finally, we use a simple memory hierarchy performance model to demonstrate how small miss rate prediction errors can result in substantial errors in the estimation of e ective access times. Trace Gathering Technique The most accurate way to collect address and data reference streams is to use a hardware monitoring device. Limitations of this technique as pointed out by Agarwal et al. [ASH86] are its complexity and limited capacity. We have built a hardware monitor which, in conjunction with limited software support, overcomes these limitations. It is capable of collecting traces which include all references generated by the CPU: user references, system call references, interrupt routine references, exception handler references, instruction prefetches, and